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SINGLE PROJECTION KACZMARZ EXTENDED ALGORITHMS 


STEFANIA PETRA, CONSTANTIN POPA 


Abstract. To find the least squares solution of a very large and inconsistent system of equations, 
one can employ the extended Kaczmarz algorithm. This method simultaneously removes the error 
term, such that a consistent system is asymptotically obtained, and applies Kaczmarz iterations for the 
current approximation of this system. For random corrections of the right hand side and Kaczmarz 
updates selected at random, convergence to the least squares solution has been shown. We consider the 
deterministic control strategies, and show convergence to a least squares solution when row and column 
updates are chosen according to the almost-cyclic or maximal-residual choice. 
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1. Introduction 

The Kaczmarz algorithm llKac37l for solving linear systems of the form 

Ax = h, A G 6 G M”" (1.1) 

in the least-squares sense is a protoypical instance of so-called iterative row-action methods BCenSllI 
that can be applied to very large systems of equations. Typical applications include image reconstruc¬ 
tion from tomographic projections IIGBH701 - see IICZ971 for an overview and further examples. The 
Kaczmarz algorithm has recently gained some renewed interest through the work IISV09i where an 
expected exponential convergence rate was shown for a randomized control scheme, used to define 
the sequence of Kaczmarz iterations. 

In view of practical applications where measurements define the vector b in (11.11) . the inconsistent 
case 

b ^ n{A) (1.2) 
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is significant due to measurement errors and noise that most likely take b outside the range 7^(A) of A. 
Needell liNeelOl extended to this case the analysis of IISV091I and showed a similar rate of convergence 
to a ball around the solution to the consistent system whose radius depends on the condition number 
of A and the perturbation of b. Throughout this paper we consider the inconsistent system 

Ax = b (1.3) 


after an error vector r is added to the “clean” right side b. 

Popa I Pop95b I introduced the extended Kaczmarz iteration so as to achieve convergence to a least- 
squares solution in the inconsistent case (I1.2I) . The basic idea is to interleave “row-actions” on x with 
“columns-actions” on b. The latter iteratively remove the spurious component of b orthogonal to TZ{A) 


r-=Pn{AAib)- ( 1 - 4 ) 

In a very recent paper IIZF13I1 a theoretical bound of the expected convergence rate was established for 
a randomized version of the extended Kaczmarz iteration. 

This line of research focusing on the convergence rate of randomized (extended) Kaczmarz itera¬ 
tions also connects to earlier work on establishing convergence of the deterministic Kaczmarz iteration 
when applied to inconsistent linear systems. The issue of cyclic convergence in this connection was 
recognized early IIGPR671 iTanVll ICEG83I but not resolved, as discussed next. 


Contribution. The present paper has the following objective: we establish convergence of the ex¬ 
tended Kaczmarz iteration for a particular control scheme - henceforth called maximal-residual con¬ 
trol scheme - used to define the sequence of iterates: at each iterative step the largest residual with 
respect to x and b determines the row- and column action to be performed as subsequent iterative 
step. It is evident that this scheme most aggressively aims to achieve convergence based on additional 
computational costs encountered when determining the maximal residuals. Convergence however 
was neither established in | Pop95b | nor somewhere else in the literature, to our knowledge. This also 
holds for the application of the almost cyclic control scheme IICZ97II to inconsistent linear systems. 
Our present work also fills this gap in the literature. 


Organization. We recall the classical Kaczmarz algorithm in Section |2l We specify in Section [3] 
different iterative schemes based on the Kaczmarz algorithm and its deterministic and randomized 
extensions discussed above. This section also includes preparatory Lemmata for the convergence 
analysis of the maximal-residual control scheme, and the almost cyclic control scheme, established in 
Section m We conclude and indicate further directions of research in Section [5] 


Notation. We set [n] = {1,. .. , n} for n E N. (•, •) denotes the Euclidean inner product and 11-11 = 
II • II2 = (•) the corresponding norm. Eor an m x n real matrix A, AJ will be its transpose and 
TZ{A), J\f{A) its range and null space, respectively. 5-*- will denote the orthogonal complement of 
some vector subspace S' C and Pc the orthogonal projector onto some closed convex set C. Eor 
given b E and A E we define the orthogonal decomposition 

b = b + r, ben{A), r Gn{A)^ =Af{A~^). (1.5) 

The set of least-squares solution to problem (11.11) is denoted by 

LSS{A]b) = {x E M”: X = xls +Af{A), Axis = P'R.(A){b) = b} (1.6) 

The probability simplex in is 

An = |x E M”: X > 0, ^ x* = l|. 

iG[n] 


(1.7) 
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||zl ||2 denotes the spectral norm of a linear mapping A : defined by 


PII2 = sup 


Pa :||2 


and II the Frobenius norm. The Moore-Penrose pseudoinverse is denoted 

by . Vectors are enumerated with superscripts x* and vector and matrix components with subscripts 
Xi, Aij. Specifically, matrix rows and columns are denofed by 

Ai (row f) and A^ (column j) (1.8) 

respectively. E[-] denofes fhe expecfafion operafion applied fo a random variable. denofes fhe space 
of all absolufely summable sequences {xk)kGn satisfying ^ while £+ will denofe 

nonnegative sequences. The space of convergenf sequences is denofed by ic, and ic^ denofes fhe 
space of sequences converging fo zero. 


2. The Kaczmarz Algorithm 

The Kaczmarz Algorifhm was firsf published llKac37ll . In if’s simples! form fhe Kaczmarz iferafion 
proceeds as follows: 


Algorithm 1 Kaczmarz (K) 


Require: A € b G kmax G N return Approximation to xls at bounded distance to xls 

(proportional to noise and condition number) 

Initialization x° G M", /cmax 
for fe = 1, . . . , fcmax do 
for ffc = 1 ,..., m do 


Set 


end for 
end for 




X = X — 




A. 


A. 


'^k ‘ 


-'Ik 


( 2 . 1 ) 


In the field of image reconstruction it is known as ART {Algebraic Reconstruction Technique) and 
independently rediscovered in IGBH701 . The algorithm is a particular Projection Onto Convex Sets 
(POCS) algorithm IIBB96I . and can also be viewed as a special instance of Bregman’s balancing 
method ||Bre65L which for each i := {k mod m) + 1 finds 

^k+i = -xY , 

where (x^) is the orthogonal projection of x^ on the i-th hyperplane 77* = {x G M”, {Ai,x) = 

k}. 

This sequential POCS method converges in the consistent case to a point in the intersection of 
the convex sets, see IIGPR671 Th. 1]. However, in the inconsistent case it does not converge, but 
convergence of the cyclic subsequences, called cyclic convergence, can be shown IIGPR671 Th. 2]. 

For the Kaczmarz algorithm (without relaxation), Kaczmarz IIKac37ll proved convergence to the 
unique solution of the system, provided A is square and invertible. Herman et al. showed in IIHLL781I 
that ART with relaxation converges in the consistent case. The case in which no (see also IIPZ041 ) 
solution exists has been considered by Tanabe llTan71L who proved convergence to a limit cycle of 
vectors. If, the relaxation parameter ujk goes to zero, the element of the limit cycle approach the same 
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vector. This has been considered by Censor et al. IICEG83L who show that the limiting single vector 
is the least squares solution that is unique provided A has full rank. 

However, in both consistent and inconsistent case no convergence rates existed in terms of matrix 
characteristics like e.g. the matrix condition number. By considering a random row selection strategy 
a first important step was made in IISV09II for the consistent, full rank case, and expected convergence 
rates where obtained in term of linear algebraic characteristics of A. The Randomized Kaczmarz 
algorithm IISV091I triggered a series a number of recent publications IlNeelOl [ENlll INT14II . The 
convergence of the the Randomized Kaczmarz algorithm was analyzed in IlNeelOl . The expected 
convergence to a ball of fixed radius centered at the least squares solution was shown, iNeelOi Thm. 
2.1 ]. This radius is proportional to the norm of the additive noise scaled by the condition number, and 
equals at most 

k{A) max (2.2) 

iG[m] 


where k{A) = P+|| 2 P||f- 

The bound (12.21) shows that the randomized Kaczmarz method performs well when the noise in 
inconsistent systems is small. The Kaczmarz method will not converge to the least squares solution of 
an inconsistent system, since its iterates always lie in a single solution space given by a single row of 
the matrix A. 

In order to overcome this problem and converge to a least squares solution we consider an approach 
first introduced by the second author in |Pop95a|, which a employs a iteratively modified right-hand 
side vector to deal with the inconsistent case. We show next that this strategy breaks the radius barrier 
of the standard method also for deterministic row and column selection strategies, as shown before in 
1ZE13I for the random choice. 


3. Single Projection Kaczmarz Extended (KE) algorithms 

Algorithm |2] extends Algorithm [U to inconsistent systems (11.11) due to perturbations b = b + r of 
the right-hand side. 


Algorithm 2 Single Projection Extended Kaczmarz (EK) 

Require: A G b G M™, kmax £ N return Approximative least-squares solution 

Initialization G M”', = b, a, uj G (0, 2); 

for k — 1,..., kyjiax do 

Select the index jk G [re] and set 


/ = /-i 

(3.1) 

Update the right hand side as 


b^ = h- /. 

(3.2) 

Select the index G [rre] and set 


. . , {x^~^, Ai,) — Vf 

r^k _ k-l A > Ifc - 

X — X UJ II 4 112 

II II 

(3.3) 


end for 


The following Lemma examines how the correction step in (13.21) affects the perturbed hyperplanes 

Hi^ = {x: {Ai ^, x) = b\^ = bi^ + - yfj (3.4) 
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in view of the unperturbed hyperplanes 

= {x: {Ai^,x) = bij. (3.5) 

Lemma 3.1. Consider Hi^ and Hi^ defined by (13.41) and (13.51) . Then 

k 

= {x + -tiC-x^ HiJ where = 5i^Ai^, 6i^ = (3.6) 

II 

Proof. Denote i := for simplicity. For x € Hi, we have {Ai,x + 6iAi) = {Ai,x) + = 

bi + Vi — yf = 6^. Thus, x + 6iAi E Hi. Conversely, choose x E Hi^. arbitrary and set x = x — 5iAi. 
Then {Ai,x) = {Ai,x) — rj + = bi + ri — yf — rj + y^ = bi holds. Consequently x E Hi. □ 

Remark 3.1. We observe that due to the initialization = 6 of Algorithmic the decomposition (11.51 ) 
and the update rule (13.11 ). it always holds that 

/ - r E 7^(A), Mk E N. (3.7) 


3.1. Control Sequences. We will consider the following basic deterministic control sequences, cf. 
BCenSlL besides randomized control sequences IISV091IZF1 31 . 

Cyclic control: Set ik = k mod m + 1, = A: mod n + 1. 

Almost cyclic control: Select ik E [m], jk E [n], such there exist integers mo, uq with 


[m] C ^ik.\-i , ■ ■ ■, ik+niQ } 

(3.8) 

[n] C ^jkJf-i, ... ,jk+no}j 

(3.9) 

for every /c E N. 

Set-based control: Select jk E [n] and ik E [m] such that 


jk = argmax|(A^/“^)|, 

(3.10) 

iG[n] 


ik = argmax (Aj, x^“^) — 6^ . 

(3.11) 


ie[m] 


Note that these sequences depend on each other through (I3.1I) - (I3.3I) . Sequence {jk)k& relates 
to largest component \\P'ii(Ai){y^)\\ of weighted by ||Af ||, whereas the sequence {ik)km 
relates to the largest distance of x^, weighted by || Aj||, to the hyperplane defined by some row 
Ai and the right-hand side if, that is updated due to (13.21) . 

Random control: Define fhe discrete distributions 


pe Ar 


Pi = 


2 ’ 
F 


i G [m], 


q E A„, Qj = 




2 ’ 
F 


j E [n] 


(3.12) 


and sample in each step k of the iteration (13.1b 


jk ~ q 

and each step k of the iteration (13.3b 


(3.13) 


ik ~ P - 


(3.14) 


Remark It.!. We note that the cyclic control is a special case of the almost cyclic control. The maximal 
residual choice is also known as remote set control IlCenSlII . 
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Algorithm 3 Randomized Extended Kaczmarz Algorithm (REK) 

Require: A € b G M™, kmax £ N return Approximative least-squares solution 

Initialization G M"", = 6, a, a; G [0,2] 

for k — 1 ,..., kiYiax tlu 

Select the index G [n] randomly according to (13.131 ) 
and set 


yk = yk-l _a{yk-\A^^)A^K 

(3.15) 

Update the right hand side as 


b^ = b-y^. 

(3.16) 

Select the index ih G m randomly accordins to (13.141) 
and set 


x'' = x'' -UJ - — -112- -^ik- 

(3.17) 


end for 


3.2. The Randomized Extended Kaczmarz Algorithm. In the recent paper II7E1 3L authors con¬ 
sidered Algorithm [3] with a random selection of the indices and and a = a; = 1. They proved 
the following convergence result along with a convergence rate. 

Theorem 3.2. For any A, b, and = 0, the sequence generated by REK Algorithm\^with 

a = oj = 1 converges in expectation to the minimal norm solution x ls of (fO . with the asymptotic 
error reduction factor 

/ 1 \ \k/2\ 

E[||i‘-1'lsII] < (1 + 2*:y.4))||iis|p, 

where k{A) = ||A+|| 2 ||A||i 7 ’ and k{A) = aijur, where (Ti > <72 > • • • > Ur > 0 are the nonzero 
singular values of A and r = rank(A). 

3.3. The MREK Algorithm. In this subsection we will show that II 74 1| from (13.61 ) decays geomet¬ 
rically for the maximal residual choice from (13.1 II ) and, in particular, that the error norms are 
absolutely summable. These results will be in turn used to establish convergence of the MREK algo¬ 
rithm in Section |4] We first collect some facts and state a basic assumption. Eor any invertible matrix 
D G M"""" we have (cf. (fThl) ) 

x£LSS{A-,b) 47 D-^x £ LSS{AD;b). (3.18) 

As a consequence, by choosing D = Diag(||A^| ..., ||A”'||“^), we may assume w.l.o.g. that 

||A^'|| = 1, j£[n]. (3.19) 

Eirst we need a preparatory result, which can be easily proved, see e.g. IIAns841 . 

Lemma 3.3. Let a G (0, 2), 5 > 0 be defined by 

5 = inf{|| II, C G AA(A^)^ = 7^(A), || C ||= 1 }, (3.20) 

and let A = UT,V~^, S = diag{ai,... ,ar,0, ■ ■ ■ ,0) be a singular value decomposition of A, where 
r = rank(A). Then 


0 < 6 = ar < CTl. 


( 3 . 21 ) 
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Remark 3.3. Since af,i& [r], are the eigenvalues of A and AA~^, respectively, scaling of A ■(— ^A 
by some factor c > 0 scales the singular values ai ^ Oijcd& well. Thus, by scaling the linear system 

O), 

II Ax — 6 11= min! 


'n . 

-Ax - b 11= mini, 

rp (y p 


we may assume w.l.o.g. that 5y/a{2 — a) < ^/n, hence 

(5^a(2 — a) 


1 - 


n 


G [0,1). 


(3.22) 


Lemma 3.4. Let a G (0,2), /c G N denote an arbitrary fixed number of iterations of Algorithm^ 
with ik selected according to the maximal residual choice (13.111) . and let G M and 7 *^. G be 
given by (13.61) . Then, 

(i) there exist M >0 and 7 G [0,1), independent on k, such that 

II 74 II < M 7 ^ (3.23) 

(ii) (II 74 lP)fcGN G £+ 

(iii) —)> rfor A: —)• 00, with r given by (fT31) . 

Proof (i) Update rule (13.281) yields 

— r = {y^~^ — f) — a{y^~^ , A^’^)A^^. 

Using y^ — r £ TZ^A), \/k, and TZ{A)-^ 3 r T Af j G [n], we compute 

11 / -rf = 11 /"^ -rf - a(2 - (3.24) 

Based on property (13.271) defining the index G [n], we upper bound 

117 - riP < II/-' - r|p - Yi (/■'. 

^ . r 1 

7 e[n] 

Exploiting again r ± A f j G [n], and Lemma [331 we obtain 

\\yk _ ^||2 < _ ,||2 _ ^( 2 ^ ^ ^yk-1 _ 

jgN 

a (2 — a) 


= ||/-i-rf(l- 


n 


A 


yk 1 _ ^ 2 


,k—l 


— r\\ 




— r 


< (1 - 11/- ,.ip. 

Thus, with y^ — r = b, 


H. II = rfrl/ - 7)7 ^ (1 - .. /m. 

||-4jJ| V re / minjgr^] ll^ill 


7=1- 


6^a{2 - 


a] 


re 


1/2 


(3.25) 

(3.26) 


and 7 G [0,1) due to (13.221 ). 

(ii) Using (13.251) . 7 G [0,1) and convergence of geometric series, we get 

fceN fceN 


1 — 7 ^ 


< 00 . 
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fiii') The derivation of tii shows that relation (13.251) is valid for everv i G \m]. Hence, since 7G [0,1), 
Ik — y^lloo < consf. 7^ —)• 0 for A: —)> oo. 


□ 


Algorithm 4 Algorithm Maximal Residual Extended Kaczmarz (MREK) 

Require: A G b G M™, kmax G N 

return Approximative least-squares solution 
Initialization G M”, = b; a, w G [0, 2] 

for k — 1 ,..., k^ax do 


Select the index jk G [n] such that 


l(^^ky"-')l 

> 

\{Afy^ kl, Vj G [n], 

0.11) 

and set 




.4^ 

II 

5b 

-1 _ 

-a{y’^-\A^^)A^K 

(3.28) 

Update the right hand side as 


II 

1 

(3.29) 

Select the index G [m] such that 





> 

\{Ai,x’^ k - ^*^1, V i G [m], 

(3.30) 

and set 




= x^-^ - 

- OJ 


(3.31) 

II 4- l|2 

II II 


end for 


3.4. The ACER Algorithm. In this section we will establish a result analogous to Eemma 13.41 for 
Algorithm |5] that corresponds to Algorithm |2] in the case of the almost cyclic index selection scheme. 
Eirst of all, related to (13.521) we introduce the notations 

vM = y -= y - 

and observe that the application ip'j is no more a projection and we have the equalities 

‘fjiy) = ii'^-a)k + <^^j){y)- (3-33) 

We will replay below Eemma 21 from |Pop95a| (see also llNat861 1 with respect to the above applica¬ 
tions. 

Lemma 3.5. For any a G (0, 2), y G j = 1,..., n the following are true 

II II < 1, 

II T’jy f - II y f = (2 - a)a(ll T’jy Ik - II y Ik)- 

We can now state the result analogous to Eemma [T4l 


(3.34) 

(3.35) 
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Lemma 3.6. Let A; > no G N denote an arbitrary fixed number of iterations of Algorithm\^with 
riQ defined by (I3.9I) . and with ik,jk selected according to the almost cyclic choice (13.81) and (I3.9I) . 
respectively. Let 5if, G M and 7 *^ G M"" be given by (I3.6I) . Then, 

(i) there exist M > 0 and 7 G [0,1), independent on k, such that 

II Hu II < Af7”, (3.36) 

with A: = n • no + /o) ^ G N, no > Zo G No, 

(ii) (II Hu lP)fceN G 

(iii) —)■ r for —>■ 00, with r given by (TBI) . 


Proof Step 1. Let A: > 0 be an arbitrary fixed fixed iteration of the algorithm ACEK, J = {1,..., n}, 
Jk = {jk+i, ■ ■ ■ ,jk+r} and (see ( I3.52I )) 

= Tju+r°---°Pju+y- (3.37) 

We will first show that it exists 7 G [0,1) such that 

II - ^aA(A^)(/) II< 7 II - PM(AT){y^) II • (3.38) 

From (13.371) we get 

yk+V ^ ^ayk^ ^ ^ ^ ^ ^ (3 39 ^ 

Let be the n x E matrix defined by 

A(fc) = coI{AP+\. .., AP+^). (3.40) 


Because the additional E — n columns of A(^k) are among the columns of the initial matrix A (see 
(El), we have 


^i^Jk)) thusP^(^Tp = Pyf^AT)- 


(3.41) 


If we define 


^kPuiA) we know that (see e.g. [Pop95a| ) 


n = n+ pm(at), nPM(AT) = pmiat)^ = o, ii n ii< i. (3.42) 

Then = 4>fc(?/^) = + PM{AT){y^), thus 

II - Tv(. 4^)(/) 11=11 i>t(/) 11=11 it (/ - PuiATtiA))) !!< 

II II • II 9* - Pu{AT){A) II. 

The set Jk\ J has at most E — n elements which are among the indices from J. It results that there 
are finitely many matrices thus finitely many applications i.e. 


7 = max II II belongs to [0,1), (3.43) 

which gives us (13.381) . 

Step 2. We will now show that it exists M > 0, independent on k such that 


fe —fc(mod r) 

II Ik II < Mf r , (3.44) 

with 7 from (13.431) . From (11.51) and (13.521) it results that — r G TZ{A),\/k > 0, i.e. Pjy{AT){y^) = 
r, VA: > 0. Thus, 

II yk+'^ _ J.. ||< 3 , II yfc _ J.. 11^ VA: > 0, (3.45) 


and recursively 

II y^^ — T ||< 7 II — r II, V/i > 1. 


(3.46) 
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For the arbitrary fixed index k >0, let /r be the integer given by 

k — A:(mod F) . 

= - - -, i.e. 

k = /rF + g, for some g G {0,1,... , F — 1} 

If we define M as 

M = max{|| - r ..., g° - r ||}, 
from (13.451 ) - (13.491 ) we get for any /r > 1 

II ^ II ~ II ^ II ^ T II ^ II' 

7^ II ~ ^ II < 

Hence 

In. - v’f I \\ yk U 

" " < - 


7*fc II . ^ 


M 


A, 


Ai 




M 


(3.47) 

(3.48) 

(3.49) 


(3.50) 

(3.51) 


which is exactly dMl, with M = „,{||Ad | } - 

step 3. Then, relation (13.361 ) holds directly from (13.441 ) and gives us also the conclusion (ii). Conclu¬ 
sion (iii) holds from (13.381) and the proof is complete. □ 


Algorithm 5 Algorithm Almost Cyclic Extended Kaczmarz (ACEK) 


Require: A G G R^,kmax G N, a / 0,a; 7 ^ 0 return Approximative least-squares 

solution 

Initialization G M”, = b; 

for k — 1 ,..., kyjiax do 

Select the index G [n] in an almost cyclic way according to (13.91 ) 
and set 


k k-1 {y^ 

yk = yk 1 _ aP, ^ ' A^'^. 

A^k 2 

(3.52) 

Update the right hand side as 


b^ = b- y\ 

(3.53) 


Select the index ik G [m] in an almost cyclic way according to (13.8b 
and set 


„k_k-i 


X = X — co- 


‘^k 


P 


A. 




end for 


(3.54) 


4. Convergence Analysis 


In order to prove the convergence of the two algorithms MREK|4]and ACEK|5l we next examine 
how the distance to any fixed least-squares solution changes. 

To this end, we denote by where Hi^ is the unperturbed hyperplane from (13.5b . 

given by 


= - w 


(A, 




X 


k-l\ 


— b. 




l|A 


A. 




I 


(4.1) 


Proposition 4.1. For any x G LSS{A; b) and for all A; G N, we have for every iterate x^ generated 
by the algorithm MREK^or ACEK\^ respectively and for any G [m] 
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(i) 

(ii) 

(iii) 


k ||2 II k ||2 I 2 II ||2 

X — X II =11 Xl — X II +UJ II 7jj, II , 


x'^ - X 11^=11 x^ ^ - X 11^ -U}{2 - Uj) 




\\A^,F 

k ™ ||2^|| k-1 _ ||2 , , .2 II ||2 


+ II 


x^ -X ir<ii x^~^ - X ir +u}^ II74 


(4.2) 

(4.3) 

(4.4) 


with from (1761) . 


Proof, (i) Choose x G LSS{A]b) arbitrarily. Then Ax = b and, in particular, x G Hi^. Since 
Hi ^,, Lemma 13.11 (see also (13.541) ') asserts x^ = x^ + ^lik ■ orthogonality relation 74 _L 
(x*—x) G Hi^, due to 74 = Sij^Ai^ (13.61) . immediately gives || x^ — x |p=|| x^ — x |p +uj‘^ || 74 |p . 
(ii) We will denote by Pf. the right hand side of (14.11) . i.e. 


x>: = p^^jx>^-^) = x^-^-io- 


U^kF 


A . 


(4.5) 


If 54 = {x : {Ai^,x) = 0} denotes the corresponding vector subspace (see (13.51) '). and because 

1 ^) 

64 = {Aii^,x) then the application Pg, (z) = z — ui satisfies 

x^-x = P^,^(x'^-i-x), (4.6) 

has similar properties with Lp'^ from (13.321) . Let also Ps^^ {z) = z — ’|p ^ 4 . Then, from Lemma 

13.51 (13.351) applied to Pf. and Ps^^ we get (by also using the fact that the projection Ps^^ is an 
idempotent operator) 


11^=11 -x) f = tu(2 - tu) ( II -x) II" - II x'^-i -X II") + 


- X ||"= a;(2 - u){Ps, (x'^"' - x), x'^"^ - x) + (1 - a;(2 - w)) || x'^"" - x ||"= 


„fc-i 


k-l 


„k-l 


x^-^ -X II" -a;(2-a;) 


(A4,x -x) 


114 


(4.7) 


'^k I 


Then, equation (14.31) follows from (14.21) and (14.71) . 

(iii) It results directly from (14.31) and the proof is complete. □ 

Remark 4.1. Proposition 14. IK iiif. together with Lemmata 13 .4 1 (ii) and l3.6l (ii) shows that the sequence 
(x^jfcgN generated by the MREK|4]or the ACEK algorithm |5] is quasi-Fejer of Type II, see BComOll 
Def. 1.1]. 

The next Eemma is a special case of Eemma 3.1 in BComOlll . The corresponding simplified proof 
is included for complefeness. 


Lemma 4.2. Let {ak)keN £ 4 ^^<7 {(3k)keN £ (■+ be two nonnegative sequences, and {ek)keN £ 
£+ n F satisfying 

Gfe+i = Oik — Pk + £k- (4.8) 

Then the following statements hold true. 

(i) (/3fc)fcGN e F. In particular {(3k)k&n G 4o. 

(ii) {ak)k&N converges. 
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Proof, (i) From (14.81) . we have Pk = — a^+i + efe. Furthermore, 

n n n n n 

/3fc = ~ Oik+l) + efc = Oo — On+l + Efc < «0 + 

/c=0 /c=0 /c=0 /c=0 k=0 

which yields J2ken(^k < «o + ZlfeeN^fc < +o°- Hence {(3k)kei>i G Now C 4o. shows (i). 

(ii) Now, both {ek)kGN G 4o and (/3fc)fcgN G 4o- By (ESJ), 

IcTfc+l ~ Clfcl = \^k — Pk\ < kfcl + \Pk\ = + /3fe, 

with (efc + /3fc)fcGN G This shows that {ak)ke¥i is a Cauchy sequenc^il Since {ak)keN G C M 
it also converges. □ 

We are now ready to prove convergence of MREK, Algorithm IH 

Theorem 4.3. Let a,u! £ (0, 2). The sequence {x^)k^fq generated by the MREK, Algorithm^ con¬ 
verges to a least-squares solution in LSS{A] b), for any starting vector G W^. 

Proof We split the proof into two parts, showing convergence of {x^)k^n, and convergence to a point 
in LSS{A; b), respectively. 

(i) Choose any x € LSS{A; b) and set 

II k ||2 Q m 2 II ||2 

Ofc+i =11 X -x||, lik = u:[2-u:) -p~|j2-’ ^ ^ II 74 II > 

The above Lemma (see also (4.7) !!!) asserts convergence of {ak)k&n and {l3k)k€N G in view 
of Ek £ due to Lemma lT4l (ii) and Prop. I4.1l (ii) respectively. In view of (13.311) . we get 

^ 2ui 

(4.9) 


x'^-x'^-i |P=u;2 


{Ai,,x^ A — bi. ^ 




2 cj 

< 7^- l^k + 2efc. 

1 — OJ 


Now + 2efc)fcgp} £ implies that {x^)keN is a Cauchy sequenc^land converges as well. 

In particular, using again (13.311) . 

((A4,x^-p 


x'^-x'^-^ ||2=u;2 




0 . 


(4.10) 


(ii) Assume that x^ —)• x. We show that x £ LSS{A, b). Fix any i £ [m]. Due to the particular 
choice of 4 in (13.111) . we have 

KA„x"-p - 6,1 - |r, - yfl < |(A,,x"-p - 6 , - (r, - 2 /f)| 

= KA,,x^-p- 6 ,^| 

ED . 1 

< |(A,„x"-p-6^J. 

Thus |(Aj,x^“^) — 6 ,1 0, due to |r, — y^\ —)• 0 by Lemma [Td] (hi) and (14.101) . respectively. 

Summarizing, we get lim^^oo \\Ax^~^ — 6 || = 0 = ||Ax — 6 ||. Thus, x G LSS{A, b). 

□ 


An arbitrary sequence {y^)keN is Cauchy, if \\y ~ 2/ II < o-k holds for all fc € N and {ak)keti G i fl £+ arbitrary. 


Indeed, 


n,-{-k 


- y"*!! = II - 4)11 < ErPP"' II4-"^ - 411 < Er=™'' “f = 


'*+'=-1 IU,J + 1 


+fc-l 


Sn ■■= EEl“I- Now (sn) keN is Cauchy since it converges due to (afc)fegnj G P ■ 
2 ^ 

Argument as above. 
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The main result concerning convergence of ACEK, Algorithm [51 is stated next. 

Theorem 4.4. The sequence generated by ACEK, Algorithm^ converges to a least-squares 

solution in LSS{A; b),for any starting vector E M”. 


Proof. Choose any x E LSS{A; b) and set 

ak+i =\\ x'" - X W'^, I3k = u}{2-u}) 




l|A 


2 II ii2 

^ II II 


I 


The proof of convergence —>• x is identically to the first part of the proof of Thm. 14.31 with the only 

difference that we have {ek)k€N £ due to Lemma lT6l (ii). Moreover 


^) ~ ^ik ^ ^ (4-11) 

holds. The selection of ik in (13.81) ensures [m] C {ik)k&N- This, together with (14.111) . implies Ax = b 
and completes the proof. □ 


5. Conclusions 

We consider an inconsistent system of linear equations and our goal is to find fhe leasf squares (LS) 
solution. If is known fhat fhe Kaczmarz mefhod does nof converge fo fhe LS solution in this case. 
In its randomized form the Kaczmarz method converges with a radius proportional the magnitude of 
the largest entry of the noise in the system. Convergence to the LS solution can be achieved if step 
lengths converging to zero are used. Unfortunately this significantly compromises convergence speed. 
A different approach is adopted by the extended Kaczmarz (EK) algorithm. In both randomized and 
deterministic forms, the methods alternates between projections on hyperplanes defined by fhe rows 
of the matrix and projections on the subspace orthogonal to the matrix range defined by the matrix 
columns. By this procedure the method iteratively builds a corrected right hand side which is then 
simultaneously exploited by Kaczmarz steps applied to a corrected system. The randomized extended 
Kaczmarz (REK) converges in expectation to the least squares solution and convergence rates can be 
obtained, as recently shown by Zouzias and Lreris. Lor deterministic control strategies however, the 
convergence was still open when alternating between row and columns updates. We close this gap by 
showing convergence to the LS solution. 
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